This invention relates generally to the reliability of data processing systems, and more particularly, to a system and method for managing upgrades of distributed data processing systems.
Computers are becoming increasingly vital to servicing the needs of business. As computer systems and networks become more important to servicing immediate needs, the continued availability of such systems becomes paramount. System availability is a measure of how often a system is capable of providing service to its users. System availability is expressed as a percentage representing the ratio of the time in which the system provides acceptable service to the total time in which the system is required to be operational. Typical high-availability systems provide up to 99.999 percent (five-nines) availability, or approximately five minutes of unscheduled downtime per year. Certain high-availability systems may exceed five-nines availability.
In order to achieve high-availability, a computer system provides means for redundancy among different elements of the system. Clustering is a method for providing increased availability. Clusters are characterized by multiple systems, or xe2x80x9cnodes,xe2x80x9d that work together as a single entity to cooperatively provide applications, system resources, and data to users. Computing resources are distributed throughout the cluster. Should one node fail, the workload of the failed node can be spread across the remaining cluster members. An example of a clustered computer system is the Sun(trademark) Cluster product, manufactured by Sun Microsystems, Inc.
Most high-availability service clusters (such as, for example, a network element in a telephone network) contain redundant components that can take over in case some components fail. Such redundant components are used not only to compensate for failed system components (xe2x80x9cfailoverxe2x80x9d), but are also used to make it possible to upgrade the system with no downtime. One preferred way to upgrade a system with redundant components is to use the so-called xe2x80x9crolling upgrade,xe2x80x9d in which nodes are taken down and upgraded one at a time. For a pair of nodes, a down node""s redundant component operates in its place while the other node of the pair is down. One drawback to this method is that it requires all pairs of nodes to be full interoperable between old and new versions of the nodes. Thus, in general, a rolling upgrade can only be used in systems that are written and specified to have full interoperability between the old version and the upgrade version. Many application developers are not prepared to make this commitment. Another upgrade mode is a xe2x80x9csplit mode upgrade,xe2x80x9d where redundant components are taken out of service, and upgraded to a new release. There is then an exchange of states from the old components to the new components. During this transition, however, old components only provide service to old components, and new components only provide service to new components. The only interoperation is the exchange of states between old and new instances of components implementing the same service. After the new components take over, the old components can be upgraded. This method generally does not entail any loss of service, but may involve a temporary loss of redundancy or capacity.
As an example, a typical split mode upgrade might involve the following steps:
identify components to comprise the new domain
fail any of these components that are busy over to spares
take the new domain components out of service
upgrade the new domain components
form a new high-availability cluster of the upgraded elements
match-make between old and new clusters
perform state-transfer and cut-over from old to new cluster
take old cluster elements out of service
upgrade old cluster elements
have upgraded old elements join new cluster
re-establish preferred primary/secondary configurations.
Each of these steps is, by itself, a non-trivial operation that can fail, and the overall process must run to completion even if the process that is driving it fails halfway through the process. Generally, system upgrades involve the user specifying in detail which components have which actions done to them at which times. This involves detailed planning on the part of the system operator and does not lend itself to changes or modification of the upgrade process
Thus, there is a need for a system that manages upgrades in a distributed processing system that is robust enough to handle failures during the upgrade process.
The present invention manages systems upgrades in a high-availability computer system by viewing the upgrade process as driving the system between a succession of stable configurations. Each of these stable configurations is a configuration in which the system can be safely left for an extended period of time without failure, if need be. The mechanism used by a described embodiment is an availability manager that is capable of ascertaining the state of each component and driving it toward a goal state by driving toward a succession of desired stable configurations.
If the availability manager fails in the middle of the process, the intermediate configuration (how far the upgrade has gotten) is of no importance because a new instance of the availability manager will simply look at the current and the desired configurations and start driving each individual component toward its desired state (and ultimately toward the system goal configuration).
If any of these operations fails (i.e., the control engine determines that it is not possible to drive the system to the desired configuration), unwinding the operation is accomplished by running backwards through the succession of configurations until an acceptable alternative configuration is reached. Thus, even if an upgrade cannot be accomplished, the availability of the system is maintained.
In one embodiment of the present invention, a high-availability computer system includes a plurality of nodes. Each node includes a plurality of components, which can be hardware or software entities within the computer system. An availability management system manages the operational states of the nodes and components.
The availability management system includes an orchestration agent that communicates with the availability manager and manages the high-level succession of configurations. When an orchestration agent decides that a target configuration has been achieved, it instructs the availability manager to start driving the components toward a the states of the next target configuration.
Advantages of the invention will be set forth in part in the description which follows and in part will be apparent from the description or may be learned by practice of the invention. The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims and equivalents.